{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 补充知识点2：新闻爬取 - 百度新闻爬虫"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "百度新闻爬虫的相关知识点在笔者的上一本书《Python金融大数据挖掘与分析全流程详解》（https://item.jd.com/12568751.html?dist=jd）\n",
    "\n",
    "第三章“金融数据挖掘案例实战1”中有详细的讲解，这里结合了本书第二章pandas库的相关知识点将数据进行整合，并导出成Excel文件方便使用。\n",
    "\n",
    "因为核心知识点在上一本书已经描述的较为详细，所以这里就不在赘述，通过如下代码可以爬取百度新闻多页多个关键词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "华能信托第1页爬取成功\n",
      "华能信托第2页爬取成功\n",
      "华能信托第3页爬取成功\n",
      "华能信托第4页爬取成功\n",
      "华能信托第5页爬取成功\n",
      "华能信托第6页爬取成功\n",
      "华能信托第7页爬取成功\n",
      "华能信托第8页爬取成功\n",
      "华能信托第9页爬取成功\n",
      "华能信托第10页爬取成功\n",
      "人工智能第1页爬取成功\n",
      "人工智能第2页爬取成功\n",
      "人工智能第3页爬取成功\n",
      "人工智能第4页爬取成功\n",
      "人工智能第5页爬取成功\n",
      "人工智能第6页爬取成功\n",
      "人工智能第7页爬取成功\n",
      "人工智能第8页爬取成功\n",
      "人工智能第9页爬取成功\n",
      "人工智能第10页爬取成功\n",
      "科技第1页爬取成功\n",
      "科技第2页爬取成功\n",
      "科技第3页爬取成功\n",
      "科技第4页爬取成功\n",
      "科技第5页爬取成功\n",
      "科技第6页爬取成功\n",
      "科技第7页爬取成功\n",
      "科技第8页爬取成功\n",
      "科技第9页爬取成功\n",
      "科技第10页爬取成功\n",
      "体育第1页爬取成功\n",
      "体育第2页爬取成功\n",
      "体育第3页爬取成功\n",
      "体育第4页爬取成功\n",
      "体育第5页爬取成功\n",
      "体育第6页爬取成功\n",
      "体育第7页爬取成功\n",
      "体育第8页爬取成功\n",
      "体育第9页爬取成功\n",
      "体育第10页爬取成功\n",
      "Python第1页爬取成功\n",
      "Python第2页爬取成功\n",
      "Python第3页爬取成功\n",
      "Python第4页爬取成功\n",
      "Python第5页爬取成功\n",
      "Python第6页爬取成功\n",
      "Python第7页爬取成功\n",
      "Python第8页爬取成功\n",
      "Python第9页爬取成功\n",
      "Python第10页爬取成功\n",
      "娱乐第1页爬取成功\n",
      "娱乐第2页爬取成功\n",
      "娱乐第3页爬取成功\n",
      "娱乐第4页爬取成功\n",
      "娱乐第5页爬取成功\n",
      "娱乐第6页爬取成功\n",
      "娱乐第7页爬取成功\n",
      "娱乐第8页爬取成功\n",
      "娱乐第9页爬取成功\n",
      "娱乐第10页爬取成功\n",
      "文化第1页爬取成功\n",
      "文化第2页爬取成功\n",
      "文化第3页爬取成功\n",
      "文化第4页爬取成功\n",
      "文化第5页爬取成功\n",
      "文化第6页爬取成功\n",
      "文化第7页爬取成功\n",
      "文化第8页爬取成功\n",
      "文化第9页爬取成功\n",
      "文化第10页爬取成功\n",
      "阿里巴巴第1页爬取成功\n",
      "阿里巴巴第2页爬取成功\n",
      "阿里巴巴第3页爬取成功\n",
      "阿里巴巴第4页爬取成功\n",
      "阿里巴巴第5页爬取成功\n",
      "阿里巴巴第6页爬取成功\n",
      "阿里巴巴第7页爬取成功\n",
      "阿里巴巴第8页爬取成功\n",
      "阿里巴巴第9页爬取成功\n",
      "阿里巴巴第10页爬取成功\n",
      "腾讯第1页爬取成功\n",
      "腾讯第2页爬取成功\n",
      "腾讯第3页爬取成功\n",
      "腾讯第4页爬取成功\n",
      "腾讯第5页爬取成功\n",
      "腾讯第6页爬取成功\n",
      "腾讯第7页爬取成功\n",
      "腾讯第8页爬取成功\n",
      "腾讯第9页爬取成功\n",
      "腾讯第10页爬取成功\n",
      "京东第1页爬取成功\n",
      "京东第2页爬取成功\n",
      "京东第3页爬取成功\n",
      "京东第4页爬取成功\n",
      "京东第5页爬取成功\n",
      "京东第6页爬取成功\n",
      "京东第7页爬取成功\n",
      "京东第8页爬取成功\n",
      "京东第9页爬取成功\n",
      "京东第10页爬取成功\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import re\n",
    "headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'}\n",
    "\n",
    "def baidu(keyword, page):  # 定义函数，方便之后批量调用\n",
    "    num = (page - 1) * 10\n",
    "    url = 'https://www.baidu.com/s?tn=news&rtt=4&bsst=1&cl=2&wd=' + keyword + '&pn=' + str(num)\n",
    "    res = requests.get(url, headers=headers).text  # 通过requests库爬虫\n",
    "    \n",
    "    # 正则提取信息\n",
    "    p_href = '<h3 class=\"c-title\">.*?<a href=\"(.*?)\"'\n",
    "    p_title = '<h3 class=\"c-title\">.*?>(.*?)</a>'\n",
    "    p_info = '<p class=\"c-author\">(.*?)</p>'\n",
    "    href = re.findall(p_href, res, re.S)\n",
    "    title = re.findall(p_title, res, re.S)\n",
    "    info = re.findall(p_info, res, re.S)\n",
    "\n",
    "    # 数据清洗\n",
    "    source = []\n",
    "    date = []\n",
    "    for i in range(len(title)):\n",
    "        title[i] = title[i].strip()\n",
    "        title[i] = re.sub('<.*?>', '', title[i])\n",
    "        info[i] = re.sub('<.*?>', '', info[i])\n",
    "        source.append(info[i].split('&nbsp;&nbsp;')[0])  \n",
    "        date.append(info[i].split('&nbsp;&nbsp;')[1])\n",
    "        source[i] = source[i].strip()\n",
    "        date[i] = date[i].strip()\n",
    "    \n",
    "    # 通过2.2.1节字典生成二维DataFrame表格   \n",
    "    result = pd.DataFrame({'关键词': keyword, '标题': title, '网址': href, '来源': source, '日期': date})\n",
    "    return result\n",
    "    \n",
    "# 通过pandas库将数据进行整合并导出为Excel\n",
    "import pandas as pd  \n",
    "df = pd.DataFrame()\n",
    "    \n",
    "keywords = ['华能信托', '人工智能', '科技', '体育', 'Python', '娱乐', '文化', '阿里巴巴', '腾讯', '京东']\n",
    "for keyword in keywords:\n",
    "    for i in range(10):  # 循环10遍，获取10页的信息\n",
    "        result = baidu(keyword, i+1)\n",
    "        df = df.append(result)  # 通过append()函数添加每条信息到df中\n",
    "        print(keyword + '第' + str(i+1) + '页爬取成功')\n",
    "\n",
    "df.to_excel('新闻_new.xlsx')  # 在代码所在文件夹生成EXCEL文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>关键词</th>\n",
       "      <th>标题</th>\n",
       "      <th>网址</th>\n",
       "      <th>来源</th>\n",
       "      <th>日期</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>华能信托</td>\n",
       "      <td>68家信托公司战略各异:头部强调综合金融、中游重差异化、下游战略...</td>\n",
       "      <td>http://baijiahao.baidu.com/s?id=16443478269695...</td>\n",
       "      <td>同花顺财经</td>\n",
       "      <td>2019年09月11日 11:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>华能信托</td>\n",
       "      <td>【调研快报】好太太接待华能贵诚信托等32家机构调研</td>\n",
       "      <td>http://stock.eastmoney.com/a/20190902122530361...</td>\n",
       "      <td>东方财富网</td>\n",
       "      <td>2019年09月02日 17:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>华能信托</td>\n",
       "      <td>「0812」信托数据研报-小貔貅出品</td>\n",
       "      <td>http://baijiahao.baidu.com/s?id=16415551320291...</td>\n",
       "      <td>小貔貅WealthTech</td>\n",
       "      <td>2019年08月12日 07:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>华能信托</td>\n",
       "      <td>信托公司2019年上半年经营业绩概览</td>\n",
       "      <td>http://www.financialnews.com.cn/trust/hyzx/201...</td>\n",
       "      <td>中国金融新闻网</td>\n",
       "      <td>2019年07月23日 14:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>华能信托</td>\n",
       "      <td>58家信托上半年净利排位!平安江苏中信华能位列前四</td>\n",
       "      <td>http://finance.sina.com.cn/trust/xthydt/2019-0...</td>\n",
       "      <td>新浪</td>\n",
       "      <td>2019年07月18日 12:14</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    关键词                                   标题  \\\n",
       "0  华能信托  68家信托公司战略各异:头部强调综合金融、中游重差异化、下游战略...   \n",
       "1  华能信托            【调研快报】好太太接待华能贵诚信托等32家机构调研   \n",
       "2  华能信托                   「0812」信托数据研报-小貔貅出品   \n",
       "3  华能信托                   信托公司2019年上半年经营业绩概览   \n",
       "4  华能信托            58家信托上半年净利排位!平安江苏中信华能位列前四   \n",
       "\n",
       "                                                  网址             来源  \\\n",
       "0  http://baijiahao.baidu.com/s?id=16443478269695...          同花顺财经   \n",
       "1  http://stock.eastmoney.com/a/20190902122530361...          东方财富网   \n",
       "2  http://baijiahao.baidu.com/s?id=16415551320291...  小貔貅WealthTech   \n",
       "3  http://www.financialnews.com.cn/trust/hyzx/201...        中国金融新闻网   \n",
       "4  http://finance.sina.com.cn/trust/xthydt/2019-0...             新浪   \n",
       "\n",
       "                  日期  \n",
       "0  2019年09月11日 11:22  \n",
       "1  2019年09月02日 17:06  \n",
       "2  2019年08月12日 07:00  \n",
       "3  2019年07月23日 14:08  \n",
       "4  2019年07月18日 12:14  "
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
